offchain processing macro and example for sov-nft-module #939

dubbelosix · 2023-09-27T12:50:52Z

Description

This PR contains the code for offchain macro which enables creating functions that can update and manage offchain state/computation that doesn't impact on-chain state or consensus.
Decorating a function with the offchain macro causes it to no-op unless the "offchain" feature is explicitly enabled. When the offchain feature is enabled while starting the rollup as described in offchain_readme.md, the code is executed.

Components added

offchain macro in sov-module-macros
offchain.rs inside the sov-nft-module for updating postgres tables in sync with transactions
init_db.sql which has the scripts to create the necessary tables for offchain processing
utils/nft-utils - a binary that can be used to submit transactions to the sequencer
simple_client exists with send_transaction, which sends a single transaction in a batch. added send_transactions which takes a vector of transactions, serializes them into a batch and submits to the sequencer. optionally, send_transactions can also take a parameter that controls the batch size. For example Some(3) would mean that multiple batches of up to 3 transactions each are submitted, and None would mean that there is micro batching, and all transactions are submitted.

Outstanding Issues

There is a timing/concurrency issue with the sequencer and the way it reads from state to accept/reject transactions, so the nft-utils script contains a sleep of 500ms between sends
sequencer also performs an execute so offchain processing is being done twice. This is not an issue for any of the tables except "top_owners" since the operation is not idempotent (increments are done twice, once in batch_builder and once more in sov_modules_stf_template. investigating options to avoid this double processing

Testing

The script added to create collections and mint NFTs at utils/nft-utils can also be used to test

Docs

Docmentation specific to using and testing the offchain macro is at module-system/module-implementations/sov-nft-module/offchain_readme.md

codecov · 2023-09-27T13:09:32Z

Codecov Report

Merging #939 (be181d5) into nightly (7229226) will decrease coverage by 1.1%.
The diff coverage is 16.6%.

Files	Coverage Δ
full-node/sov-stf-runner/src/batch_builder.rs	`98.0% <100.0%> (+<0.1%)`	⬆️
.../module-implementations/sov-nft-module/src/call.rs	`97.3% <100.0%> (+0.1%)`	⬆️
...m/module-implementations/sov-nft-module/src/lib.rs	`96.4% <ø> (ø)`
...ule-implementations/sov-nft-module/src/offchain.rs	`100.0% <100.0%> (ø)`
module-system/sov-modules-api/src/transaction.rs	`100.0% <100.0%> (ø)`
module-system/sov-modules-macros/src/lib.rs	`100.0% <100.0%> (ø)`
module-system/sov-modules-macros/src/offchain.rs	`100.0% <100.0%> (ø)`
full-node/sov-sequencer/src/utils.rs	`56.8% <0.0%> (-40.2%)`	⬇️
utils/nft-utils/src/main.rs	`0.0% <0.0%> (ø)`
utils/nft-utils/src/lib.rs	`0.0% <0.0%> (ø)`

... and 1 file with indirect coverage changes

full-node/sov-sequencer/src/utils.rs

module-system/module-implementations/sov-nft-module/Cargo.toml

module-system/module-implementations/sov-nft-module/offchain_readme.md

module-system/module-implementations/sov-nft-module/src/call.rs

module-system/module-implementations/sov-nft-module/src/init_db.sql

module-system/module-implementations/sov-nft-module/src/offchain.rs

module-system/sov-modules-macros/src/lib.rs

utils/nft-utils/src/main.rs

utils/nft-utils/Cargo.toml

utils/nft-utils/src/main.rs

* implement eth_call * implement tests and error handling for eth_call * move errors inside evm crate * cleanup result * fix call env * comment test differences * rebase * Revert "rebase" This reverts commit 44e41b2. * fix unused imports * fix review notes * use mix_hash as prevrandao * improve test s

This reverts commit d443009.

* Remove demo-rollup dep from demo-prover * add methods to demo-rollup * fix cargo hack * fix cargo hack * Cargo.toml * install toolchain * install toolchain * modify workflow * update cargo.toml * rust.yaml * Add guest-mock * use real prover in bank tests * native * fix lint * update yaml * fic coverage * ELF in CI * update bench get_guest_options() * update build.rs * remove local from demo-rollup * fix lint * remove ide_setup * Remove riscv32im-risc0-zkvm-elf

* fix: require `native` when applicable for all targets Prior to this commit, some checks of all-targets without the feature `native` would break. This commit introduces a fix for every workspace member to be consistent with the feature set. * change feature requirement to self-dev-dep * update deps * fix dupl celestia dependencies

* Make genesis config serializable * Evm config serde * Fix chain state integ test * fix vec-setter

* Add PublicKeyHex * PubKeyHex impl * Update DefaultPublicKey::from_str * Add tests * Remove println * Add doc

* implement account endpoints * test account endpoints * remove unnecessary conversions

* feat: add gas meter to working set This commit introduces `GasMeter`, encapsulated by `WorkingSet`. It will allow the user to consume scalar gas from the working set, and define arbitrary price parsed from a constants.json manifest file at compilation. At each compilation, the `ModuleInfo` derive macro will parse such file, and set the gas price configuration. * fix lint fmt * fix ci test expected error string * update default context to 2 dimensions

* PrivateKeyHex in accounts * Add test_config_serialization * Add accounts.json * fix ci * fix CI * cleanup * Arbitrary PublicKeyHex * PublicKeyHex impl

dubbelosix requested review from bkolad, citizen-stig and preston-evans98 September 27, 2023 12:50

dubbelosix marked this pull request as ready for review September 27, 2023 13:32